SPLUNK


Outline

Module 1: Introducing Splunk Module 2: Splunk Components Module 3: Installing Splunk Module 4: Getting Data In Module 5: Basic Search Module 6: Using Fields Module 7: Best Practices

Module 8: Splunk’s Search Language Module 9: Transforming Commands

Module 10: Creating Reports and Dashboards Module 11: Pivot and Datasets

Module 12: Creating and Using Lookups

Module 13: Creating Scheduled Reports and Alerts



Module 1 Introducing Splunk


Understanding Splunk


What Is Splunk?


Aggregate, analyze, and get answers from your machine data


Index ANY data from ANY source


• Databases

Note

For lots of ideas on data to collect in your environment, get the Splunk publication The Essential Guide to Machine Data.



Splunk

Search Head

Splunk

Indexer


Forwarder



Search & Reporting App (cont.)

splunk bar

current app


app navigation bar


current view


search bar


global stats


time range picker

start search


data sources


search history


Click Data Summary to see hosts, sources, or sourcetypes on separate tabs



Deployer Index Cluster



Module 3: Installing Splunk


There are multiple Splunk components installed from the Splunk Enterprise package



Splunk Enterprise

Indexer (Search peer)

Search Head Deployment

Server

License Master

Heavy Forwarder

Cluster Master

Search Head Cluster



3


For ongoing monitoring

For one-time indexing (or testing); the Index Once option does not create a stanza in inputs.conf


Set Source Type (Data Preview Interface)

1

3

4


2


1

Splunk automatically determines the source type for major data types when there is enough data

2

You can choose a different source type from the dropdown list

3

Or, you can create a new source type name for the specific source

4

Data preview displays how your processed events will be indexed

If the events are correctly separated and the right timestamps are highlighted, you can move ahead

If not, you can select a different source type from the list or customize the settings



  1. Search History displays your most recent ad-hoc searches – 5 per page


  2. You can set a time filter to further narrow your results

    1


    2

  3. Click the > icon in the leftmost 3

column to expand long queries to display the full text




Module 6:

Using Fields in Searches


              What    Are    Fields?           


indicates the field’s values are alphanumeric


indicates that the majority of the field values are numeric



Make an Interesting Field a Selected Field

2

1



  • Example: status != 200


Returns events where status

equal 200

field

exists

and

value

in field

doesn’t







Returns events where status

field

exists

and

value

in field

doesn’t

equal 200 -- and all events where status field doesn’t exist


In this example:


  1. Search terms – what are you looking for?

  2. Commands – what do you want to do with the results?

  3. Functions – how do you want to chart, compute, or evaluate the results?

  4. Arguments – are there variables you want to apply to this function?

  5. Clauses – how do you want to group or rename the fields in the results?



Disk

Intermediate results table

Intermediate results table

Final results table


index=security sourcetype=linux_secure fail* | top user | fields – percent



Fetch events from disk that match


Summarize into table of top 10 users


Remove column showing percentage



| table clientip, action, productId, status

| rename productId as ProductID, A

action as "Customer Action", B

status as "HTTP Status"

C

Renaming Fields in a Table



fields Command – Examples

Using command improves performance—only specified fields extracted

Scenario

Display network failures during the previous week.

Returned 6,567 results by scanning 6,567 events in 1.425 seconds:


index=security sourcetype=linux_secure (fail* OR invalid)


Scenario

Display network failures during the previous week. Retrieve only user, app, and src_ip.

Returned 6,567 results by scanning 6,567 events in 0.753 seconds:


index=security sourcetype=linux_secure (fail* OR invalid)

| fields user, app, src_ip


Use dedup to remove duplicates from your results

index=sales sourcetype=vendor_sales Vendor=Bea* | table Vendor, VendorCity, VendorStateProvince, VendorCountry


…| dedup Vendor | table … …| dedup Vendor, VendorCity | table



limit=20

results, use the limit option


... | sort

... | sort


20 count

–categoryId, product_name


sort -/+<fieldname> sign followed by fieldname sorts results in the sign's order

sort -/+ <fieldname> sign followed by space and then fieldname applies sort

order to all following fields without a different explicit sort order


index=sales sourcetype=vendor_sales Vendor=Bea*

| dedup Vendor, VendorCity

| table Vendor, VendorCity, VendorStateProvince, VendorCountry

| sort –Vendor, VendorCity

index=sales sourcetype=vendor_sales Vendor=Bea*

| dedup Vendor, VendorCity

| table Vendor, VendorCity, VendorStateProvince, VendorCountry

| sort – Vendor, VendorCity





139


Three main methods to create tables and visualizations in Splunk are:

  1. Select a field from the fields sidebar and choose a report to run

  2. Use the Pivot interface

  3. Use the Splunk search language transforming commands in the Search bar



    Creating Reports From the Field Window

    This is known as a

    timechart


    Creating a Top Values Report


    Creating a Top Values Report (cont.)

  4. The top command with limit=20 is added to the search string

  5. A bar chart is returned on the Visualizations tab, displaying the top categories purchased






Adding a Report to a Dashboard

In the report, click Add to Dashboard to begin




A


B


C


D


E

Adding a Report to a Dashboard (cont.)

A

Name the dashboard and optionally provide a description

B

Change the permissions (use Private until tested)

C

D

Enter a meaningful title for the panel For Panel Powered By, click Report

E

For Panel Content, click Column Chart to display the visualization in the dashboard


Note

The Dashboard ID is automatically populated with a unique value used by Splunk and should not be changed.


Adding a Report to a Dashboard (cont.)

After it is saved, you can view the dashboard immediately, or select the dashboard from the Dashboards view





Click on the dotted bar on a panel to drag the panel to a new location


More Options icon (discussed on next slide)


  1. In Edit Dashboard mode, click the More Options icon on any panel and select Edit Drilldown

  2. In Drilldown Editor, select Link to search to access search directly from visualization


More Options icon


1 2


Once drilldown option is set, click an object in a chart or table to see its underlying events in Search view



  1. Click the ellipsis menu (...) and select Clone

  2. Change the Title as desired and click Clone Dashboard


    1


    2


    Dashboards can be exported as PDF or printed





After you’ve set a dashboard as default, your home view may look like this:



Module 11 Pivot & Datasets


  1. From the Search & Reporting app, select the Datasets tab

  2. Click Explore > Visualize with Pivot




categories

count by

category

To format the results, click here


For example, to add totals on the Summary tab, click Yes next to Totals



2


1



under Split Columns and select the desired split

the maximum number of columns and whether you want Totals


1

2

3


The ALL column shows row totals by category


173 Splunk Fundamentals 1


1

2

5

4

3

6

  1. Execute a search (search criteria only, no search commands)

  2. Click the Statistics or Visualization tab

  3. Click the Pivot icon

  4. Select the fields to be included in the data model object

  5. Create the pivot (table or chart)


1


2


3 4




1


Saving a Pivot as a Report


2


3


  1. Upload the file required for the lookup

  2. Define the lookup type

  3. Optionally, configure the lookup to run automatically


    1

    2

    3


    Adding a New Lookup Table File

    Settings > Lookups > Lookup table files

    1. Click New Lookup Table File

    2. Select a destination app

      2

      3

      4

      5

    3. Browse and select

      the .csv file to use for the lookup table

    4. Enter a name for the lookup file

    5. Save


      inputlookup Command

      • Use the inputlookup command to load the results from a specified static lookup

      • Useful to:

        • Review the data in the .csv file

        • Validate the lookup

lookup definition name.


Settings > Lookups > Lookup definitions

6


2

3

4

5

  1. Click New Lookup Definition

  2. Select a destination app

  3. Name the lookup definition

  4. Select the lookup type, either File-based or External

  5. From the drop-down, select a lookup file

  6. Save




Scenario

Calculate the sales for each product in the last 24 hours.


Settings > Lookups > Automatic lookups

  1. Click New Automatic Lookup

  2. Select the Destination app

  3. Enter a Name for the lookup

  1. Select the Lookup table definition

  2. Select host, source, or sourcetype to apply to the lookup and specify the name




    2

    3


    4


    5


    Creating an Automatic Lookup (cont.)


  3. Define the Lookup input fields

    Field(s) that exist in your events that you are relating to the lookup table

    1. Column name in CSV

    2. Field name in Splunk, if different from column name

  4. Define the Lookup output

    A


    column name in lookup file

    C

    B


    field name in Splunk


    D

    fields

    Field(s) from your lookup table that are added to the events

    1. Field name in lookup table

    2. Name you want displayed in Splunk; otherwise it inherits the column name

  5. Save


To use an automatic lookup, specify the output fields in your search




  1. Create your search

  2. From the Save As menu, select Report


  3. Enter Title

  4. Enter Description

  5. Set Time Range Picker to No

  6. Click Save



    Note

    Time Range Picker cannot be used with scheduled reports.



Note

Depending on the permissions granted to you by your Splunk administrator, you may be able to set permissions to share your scheduled report.


Creating a Scheduled Report – Define Schedule


Creating a Scheduled Report – Send Email

  1. Enter addresses in the To field, separated by commas

  2. Set the priority

  3. Edit or keep the default subject

    The $name$ variable includes the name of the report

  4. If desired, include other options, such as an inline table of results

  5. Define the email text type

  6. Click Save


The proper permissions from your Splunk administrator are required to edit the permissions on a scheduled report.

Note

Managing Reports – Edit Permissions


Managing Reports – Edit Permissions (cont.)